EpiForecasts group (https://epiforecasts.io):
Akira Endo, Alexis Robert, Ciara McCarthy, Hannah Choi,
Joel Hellewell, James Azam, James Munday, Kath Sherratt,
Liza Hadley, Manuel Stapper, Nikos Bosse, Sam Abbott,
Sophie Meakin, Toshiaki Asakura
Collaborators at LSHTM and elsewhere.
Models are a tool to combine data (what we know) with assumptions and theory (what we think) to learn about what we don’t know.
When data is abundant, models and analytics can generate insight without many additional assumptions.
When data is sparse (e.g. early in an outbreak), modellers need to make more assumptions to generate insights.
January 2020: Can COVID-19 be controlled by contact tracing?
Probability of control depends on intensity of transmission and contact tracing effort.
“We illustrate the potential impact that flawed model inferences can have on public health policy with the model described […] by Joel Hellewell and colleagues, which is part of the scientific evidence informing the UK Government’s response to COVID-19.”
“All models are wrong, but some are useful”
– George Box
“All models are wrong, but some are useful”
– George Box
wrong: how wrong?
some: which ones?
Anticipate healthcare demand
Design interventions
Plan clinical trials
“We were losing ourselves in details […] all we needed to know is, are the number of cases rising, falling, or levelling off?”
– Hans Rosling, Liberia, 2014
“maximise sharpness subject to calibration”
Ebola forecasts could be trusted for up to 2 weeks
Our Ebola forecasts could be trusted for up to 2 weeks
We can compare forecasts using
proper scoring rules
\[\mathrm{CRPS}(F, x) = \mathbb{E}|X-x| - \frac{1}{2}\mathbb{E}|X-X'|\]
We can compare forecasts using proper scoring rules \[\mathrm{CRPS}(F, x) = \mathbb{E}|X-x| - \frac{1}{2}\mathbb{E}|X-X'|\] but these only tell us about relative quality of forecasts
Observed behaviour as predictor: improvement of forecasts, but only once age-specific reporting is taken into account.
Humans better than models at predicting cases, but not deaths
Amongst models, ones that focus on a single country tended to do better
Sherratt et al., work in progress
Alternative ways of measuring predictive performance change the ranking of models.
– Hans Rosling, Liberia, 2014